Analyzing NIC Overheads in Network-Intensive Workloads

Keeping up with modern high-bandwidth networks is a significant challenge for system designers. A key obstacle to high network throughput is the high overhead of communication between the CPU and network interface controller (NIC), which typically resides on a standard I/O bus with high access latency. We investigate the impact of this overhead by analyzing the performance of hypothetical systems in which the NIC is more closely coupled to the CPU, including integration on the CPU die, using three network-intensive benchmarks. We find that systems with high-latency NICs spend a significant amount of time in the device driver. NIC integration can substantially reduce this overhead, providing significant throughput benefits when other CPU processing is not a bottleneck. NIC integration also enables cache placement of DMA data; this feature has tremendous benefits when payloads are touched quickly, but can harm performance in other situations due to cache pollution.